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DYNAMIC INTERFACE SYNTHESIZER 
FIELD OF THE INVENTION 

The present invention relates to computer interfaces, and in particular to the 
5 dynamic generation of interfaces matching the capabilities of the computer to the 
information to be communicated. 

BACKGROUND OF THE INVENTION 

Interactions between humans and computer systems began with very primitive 
10 interfaces, such as a light emitting diodes and switches. The interfaces have quickly 
evolved to the use of keyboards, touchpads and joysticks for the user to communicate 
with the computer to multimedia presentations by the computer to communicate to the 
user. User interfaces also refer to the manner in which programs communicate with 
each other. Most of these interfaces are static in nature, and are supported by 
1 5 programming in applications and other programs that are running on the computer. 
When the computer is being used to control real time processes, such as building 
automation or industrial process control, it is difficult to anticipate all of the potential 
user interactions with a computer, and to provide a static interface for each such 
interaction. The process being controlled can do things that are not anticipated, and 
20 also, further equipment, such as sensors and motors can be added to the process. There 
is a need for a flexible interface that can provide for user friendly interactions with the 
computer, even when the computer is being used to control changing processes. 

Static design of user interfaces requires the application of specialized, ever- 
changing knowledge possessed by talented, trained individuals, responding to 
. 25 evolutionary and revolutionary changes in technological capabilities in an environment 
where it is difficult to quantify procedures or results. It is difficult for a programmer to 
keep up with such individuals. Once a static interface for a system is designed, using 
well known design rules, it is difficult to change, and unless there is a great need, the 
design usually remains unchanged. This can result with a system having a less than 
30 optimal user interface for extended periods of time, while the underlying capabilities of 
the system may have dramatically changed. 
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Further, even when performed optimally, static interaction design is no longer 
sufficient to' meet the task and information needs of current systems. Current systems 
are getting more complex, with multiple collaborating computational and human agents. 
Areas of maximum unpredictaliility include Which interactions will be required, what 
5 information will be available, what hardware and software capabilities will exist in local 
and distributed forms, what experience and capabilities will be contributed by humans, 
and what different devices will be used by the sarhe human at different times. This 
unpredictability will also vary extensively over time and location. 

While automatic design critiquing systems exist, there are as yet no systems 
10 which perform even the automatic design of static interactions. Earlier efforts 

abandoned design altogether, given that it was difficult to automate a design process 
that is difficult even for humans to perform. Instead, researchers concentrated on 
computer-aided design, which helps designers create static designs. Further, in US 
Patent No. 5,550,971 to Brunner et al., a knowledge base was constructed for automatic 
15 generationof query templates based on well understood and unchanging stereotyped 

interactions, such as query of a database. 

SUMMARY OF THE INVENTION 

A user interface is dynaniically synthesized for an application running on a 
20 computer system. Objects in the application both serve and create user interaction needs 
that are used by an interaction facilitator to traverse a framework of interaction elements 
and match the needs to appropriate elements by a specialization process. Abstract 
interaction elements have multiple concretizable elements that are selected by the 
interaction facilitator for use in generating an interaction space which contains elements 
25 representative of the information needs of the current users and instantiated objects in 

the application. Finally, a number of software modules are used to convert the elements 
in the interaction space to generate a user interface as a ftmction of the resources and 

devices available for the user. 

In one embodiment, muUiple generic design elements are selected as a ftinction 
30 of selected properties of an object in the application. First, a general model of 
interaction requirements is generated for general situational components of the 
application program. This model is maintained in the interaction space. Then a 
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representation of current interaction requirements is constructed using the model. 
Interaction artifacts are then created from the representation based on abstract 
compositional constraints. Finally, the user interface of the application program is 
dynamically created based on the interaction artifacts and available devices by the use of 
5 software modules referred to as technicians. A JavaTech module is used to create 

graphical user interfaces, and a SpeechTech module is used to create audio interfaces for 
devices such as telephones from the same interaction space. 

In one embodiment, the dynamic interface generator is applied to a search and 
rescue application. Dynamically changing task and display requirement are used to 
10 compose on-line designs of user interfaces. The interfaces are generated in direct 

response to the properties of the data and the user. The interaction space is inherently 
dependent on the situation The user interface is therefore, centered on the user and the 
task, and firmly rooted in the principles of good user interface design as they are 
represented in the general model. 
1 5 The dynamic interface generator benefits both developers and user of mixed 

initiative assistive systems. Developers are assured good design and a consistent look 
and feel to applications, and a large measure of platform independence. Product 
developers can rapidly and easily modify and create user interfaces to their system 
through modification of the dynamic interface generator. This helps reduce 
20 development costs and effort. Because of the ease of modification, incremental 

advances in computing capabilities, languages and infi-astructures can quickly be used to 
enhance products. User interface designers also benefit, since incremental advances in 
the state of the art can easily be added to existing general models or technician 
procedures. 

25 A fiirther advantage of the invention is that users are not required to perform 

laborious interface definition and configuration to specialize an application to their 
requirements. The interface definition and configuration is automatically provided 
along with well designed state of the art user interfaces. Users also feel more in control 
and connected to their tasks with a consistent interface over varying situations. 

30 Upgrades and model evolution are provided with dynamic responsiveness to context, 
task, situation and user intentions without the need for retraining and support. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of a computer system having a user interaction 

generator constructed in accordance with the present invention. 
Figure 2 is a block representation of a user interaction space generated by the user 

interaction generator of Figure 1 . 
Figure 3 is a block diagram representation of functional components of the user 

interaction generator of Figure 1. 
Figure 4 is a block diagram representation of compositional levels in an 

interaction space of the user interface generator of Figure 1 . 
Figure 5 is a block diagram representation of the compositional levels of a multi 

level interaction class model. 
Figure 6 is a block diagram representation of a component compositional level of 

specialization in the class model of Figure 5. 
Figures 7 A & 7B are a block diagram representation of one segment of the 
5 ^ component hierarchy showing "composed-of relationships. 

Figure 8 is a block diagram representation of one segment of the subtask 

compositional level of specialization. 
Figures 9A & 9B are a block diagram representation of a run time specialization of a 
component. 

10 Figure 10 is a block diagram representation of the class model diagram of 

interaction management authorities. 
Figure 11 is a block diagram representation of the application data model. 
Figure 12 is a block diagram representation of steps performed in the first three 
compositional levels of Figure 4 during generation of an interaction 
25 space. 



30 



DESCRIPTION OF THE EMBODIMENTS 

In the following detailed description, reference is made to the accompanying 
drawings which form a part hereof, and in which is shown by way of illustration specifu 
embodiments in which the invention may be practiced. These embodiments are 
described in sufficient detail to enable those skilled in the art to practice the invention, 
and it is to be understood that other embodiments may be utilized and that structural, 
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logical and electrical changes may be* made -without departing from the spirit and scope 
of the present invention. The following detailed description is, therefore, not to be taken 
in a limiting sense, and the scope of the present invention is defined by the appended 
claims. 

Numbering in the Figures is usually done with the hundreds and thousands digits 
corresponding to the figure number, with the exception that the same components may 
appear in multiple figures. 

The invention will be described with reference to a bock diagram of a computer 
system in Figure 1, followed by an example of a dynamically generated user interaction 
representation in Figure 2. The remaining description will describe logical 
representations of applications, and the frameworks used to generate user interaction 
spaces or interfaces on a real time dynamic basis.. 

The present invention is described with respect to an embodiment using an 
object oriented architecture. Every object (an encapsulated collection of data and/or 
programming) has an interface that defines the operations that may be performed on the 
object. Some examples of object oriented languages include C-H-, Smalltalk and Java, 
as well known to those skilled in the art. Objects are arranged in classes. Where an 
object fits into the classes defines some of the things it can do, and things that can be 
done to it. By using the class relationships and a hierarchy of user interaction elements 
the present invention provides a dynamically generated user interface. 

In Figure 1, a typical personal computer is shown generally at 1 10. A processor 
1 12 is coupled to a host bus 114. Also coupled to the host bus is a system controller 
1 16, a cache 1 1 8, a main memory 120 and a data path chip 122. Processor 1 12 is 
typically a microprocessor. Cache 1 1 8 provides high-speed local -memory data for 
processor 112 and is controlled by system controller 1 16 in a well known manner. Main 
memory 120 is coupled between system controller 1 16 and data-path chip 122, and in 
one embodiment provides random access memory of between 16 MB and 128 MB of 
data. In one embodiment, main memory 120 is provided on SIMMS (single in-line 
memory modules), which plug into suitable sockets provided on a motherboard holding 
many of the components shown in Figure 1 . 

The system controller 1 16 is also coupled to a bus 128, which represents 
multiple buses, including a PCI (Peripheral Component Interconnect) bus and an ISA 
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bus, which couple multiple other devices^ including secondary storage devices indicated 
at 130, network adapters indicated at 132 a keyboard/mouse controller 140 and a 
graphics controller indicated at 142. The keyboard controller 140 is provides a port for 
coupling to a keyboard 144, which the graphics controller 142 provides a port for 
5 coupling to a display device or monitor 146. The keyboard controller 140 and graphics 
controller 142 thus provide at least some of the user interaction capabilities that are 
dynamically utilized to allow the user to interact with and control the computer, which 
in turn is ruxming the application. The application may comprise a society of objects 
whose capabilities and needs are tracked by a dispatcher, and may make use of multiple 
10 computers which are coupled via the network card 1 32 to exchange information via an 
ethemet or other type of protocol. 

Figure 2 shows one interface or interaction space generated for a search and 
rescue situation (SARSIT) application which keeps track of and coordinates massive 
search operations for a downed pilot during wartime or other situations. The interface is 
15 used to collect and display data on dovmed aircraft, and comprises a window indicated 
generally at 210 for display on monitor 146. It should be noted that the present 
invention would dynamically create different displays on monitors having different 
capabilities, such as a monitor having touch screen ability. Further, other devices could 
be used, such as a telephone. The ''display" elements would then perhaps take the form 
20 of speech synthesis, with data entry either being by voice or telephone keypad. 

The application is composed of objects related to each other in a framework or 
class hierarchy. Many of the objects provide information related to the search 
operation, such as real time sensing of weather and tracking information derived from 
multiple automated and manual sources. In such an object oriented program 
25 envirorunent, each object contains information about the types of data it provides and 
receives. For example, an application object representing a particular lost airplane 
contains information about its properties and subparts; the application object 
representing this plane's color contains information about the current color and 
allowable colors. 

30 It is this information and placement in the framework for each object requiring a 

human interface, along v^th a related framework of interaction elements that allows the 
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dynamic generation of an internal user interface, which is further used to generate an 
actual interface dependent on interface device characteristics. 

A typical search and rescue operation application provides and processes 
information that is extremely situation specific. For example, if a pilot ejects over 
5 water, ocean current information is crucial, but if the pilot ejects over land, the ocean 

currents become irrelevant. Given the non deterministic nature of airplanes going down, 
or ships being sunk, all of the useful information sources cannot be known a priori. For 
- example, witnesses cannot be known before an incident occurs. The tactics used to 
accomplish the tasks are also situation dependent. For example, in a combat situation, 
10 searching must be accomplished by means other than flying search patterns because the 
searcher's predictability makes the search craft vulnerable to enemy fire. Crucial 
information is often not on line, making offline portions of tasks very important. For 
instance, personal pilot information and photographs might only be in hard copy format. 
Given the variability and evolution of the situation dependent tasks, the user may not be 
15 aware of many on line information sources. Finally, and most importantly, time is of 

the essence. The chances of recovering a survivor are inversely proportional to the time 
it takes to find them.^ Interfaces must be quickly adapted to each situation to provide for 
optimal interaction between the user and the system. 

Some of the elements of the display of Figure 2 comprise a list of aircraft 
20 involved in the search at 2 12 which are arranged in a hierarchical format, much like a 
folder and subfolder type of arrangement. Two edit blocks indicated at 2 1 4 and 2 1 6 
provide space for entering time of day and activity location information. An indicator 
button arrangement at 218 provide push buttons for user selection indicating the type of 
event being displayed or information being entered. The altitude of the aircraft is 
25 indicated at 220 in an edit box, with a pair of push buttons for changing the value either 
up or down. A color selection button is provided at 222, and camouflage selection 
options are provided at 224 in the form of butt:ons. An edit box is provided at 226 for 
entering the tail letters and side number of the aircraft. A cause of loss information box 
is provided at 230 along with more selection buttons for scrolling between possible 
30 listed causes. Finally, at 232, an edit box with more value changing pushbuttons is 
provided. 
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As can be seen, there . are many different types of elements that can be arranged 
on a display. In prior devices, user interface designers would provide an arrangement of 
elements for each particular desired view of information and information input 
requirements. There are many design rules and guidelines for providing the elements in 

5 a clean and orderly manner. However, as previously indicated, the application and the 
information processed by the application may rapidly change. There is little time left to 
provide an optimal design in accordance with the rules and guidelines. The present 
invention solves this problem by matching a hierarchy of design elements with the 
application and dynamically creating the user interface. 

10 Programming code to accomplish dynamic interface generation (DIG) is shown 

in a functional block diagram in Figure 3 indicated generally at 310. A specialized 
group of software agents or modules is represented and referred to as an interaction 
management team. The interaction management team 310 has the ability to 
dynamically generate concise, context-sensitive, well-designed user interfaces (UIs) 

15 between a user and a computer. to provide not only .ongoing task support, but also 
collaborative dynamic dialogs between a human and software agents. The team 
provides two basic processes comprising the generation and maintenance of an on-going 
interaction space 312, representing the interaction from the human user' s perspective. 
An interaction facilitator agent (InFac) shown at 314 performs this ftinction. The 

20 second basic process comprises the generation and maintenance of a required 

appropriate realization of the interaction space on the available hardware, using the 
available operating system and an interaction toolkit such as Java, HTML, Motif, etc as 
indicated at 3 1 5. This is also performed by interaction techs, which include many 
modules for converting the interaction space into constructs perceivable by the user. 

25 Two such techs are indicated at 3 16 and 3 1 8 as the Java Tech and the Speech Tech. 
These ftinctions allow the interaction management team to dynamically generate the 
necessary user interface 320 dictated by the current situation, the user's goal and 
capabilities, and the available hardware and software. 

The interaction management team uses a process called dynamic interaction 

30 generation (DIG) 330 to translate the actions and knowledge of the software agents into 
easily understandable representations for the human, and vice versa. DIG incorporates 
the current state of the art in user interface and software design methodologies and 
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processes to generate user interfaces at run-time. Additionally, DIG facilitates 
incorporation of advances in the state of the art or improvement to usability without 
requiring laborious modifications. 

At the heart of the DIG process is an interaction class model v^hich contains a 
5 detailed model of user interaction components and their mutual compositional 

constraints. This model is based on abstracted principles of best practices user interface 
design (Galitz, Wilbert O. The Essential Guide to User Interface Design, New York: 
Wiley, 1997, and Mandel, Theo, The Elements of User Interface Design. New York: 
Wiley, 1997., both of which are hereby incorporated by reference for the rule based 
10 knowledge that they provide in terms of interface design). The model is also based on a 
consistent approach for interaction with building management and process control 
applications such that they are intuitively operated by users, contain a consistent 
physical layout metaphor and use consistent object grouping metaphors which make 
sense given the processes being controlled. It also incorporates multiple views of 
15 information and gives users the ability to control the amount and type of information 
available through direct manipulation interaction such as by use of a mouse. 

The interaction class model has concrete classes that are instantiated, modified 
and destroyed dynamically as interaction needs change to create the interaction space 
312, which represents the on-going state of the human interaction with the computer. 
20 The model is essentially a description of the composition of various interaction levels 
that can be described for any user interface to a complex process. 

Also shown in Figure 3, is an application technician 340 which is software 
which facilitates the transfer of data between the application in the interface facilitator 
314. Further, an application domain model 342 is representative of InFac 314 
25 knowledge of the structure of the domain model of the application. 

Compositional levels of the multi level interaction class model are shovm block 
form in Figure 4. A top level 410 comprises the user interface application 
(UI Application), which represents a collection of tasks that are required by a human 
participating in a semi-automated mixed initiative collaboration, e.g., managing the 
30 environment, security, and mechanical resources of a building, or conducting a combat 
search and rescue operation. 
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An application is composed of various user, interface tasks (UITask) 412. There 
are three basic types: ' 

1 . global tasks, which involve meta-interactions such as choosing the task to 
perfomiv choosing the. object to investigate, alerting the user to the need for a 

5 system initiated interaction, 

2. object specific tasks, which are interactions with a specific object in the 
application domain such as a schedule for a particular object, the trend of a 
sensed value, or the layout of a physical space, and 

3. transient tasks, which are user- or system-initiated discourse acts such as the 

IQ system asks the user to authorize an action or the user asks the system to overlay 

town names on the map. 

User interface tasks are themselves composed of the third level of organization 
in the model: User Sub Task (UISubTask) at 414. For example, there are two 
interactions required to choose an object in the application domain: 1) Choose the 
15 information organization such as a process vs. function) and 2) Choose the object within 
the organization such as a specific report, a piece of equipment, or a plan. 

Subtasks are in turn composed of user interface components (UlComponent) 
416, which are roughly analogous in function and complexity to conunon user interface 
widgets such as value slider, trend graphs, entry boxes, and menu lists. UlComponents 
20 are not, however, directly tied to widgets; rather they are an abstract representation of 
the interactions and information rendered by those widgets. For example, a bus 
schedule is represented in the interaction space as an element from an itemovertime (see 
820) class. If it is eventually presented on a display screen at a kiosk at the bus station, 
it would be represented by a different widget such as a trend plot than it would be if it is 
25 delivered over a telephone - a series of times and stops, presented sequentially in spoken 
English. Despite their eventual mapping to very different user interface widgets, the 
elements required to convey the time-event pairs that make of the bus schedule in the 
interaction space is the same. 

The interaction space representation allows the interaction team to generate fully 
30 specified descriptions of the interaction with the human, without reference to any 

specific hardware. This type of adaptability to a current situation allows InFac 3 14 to 
make use of whatever resources are currently available, most appropriate, or preferred 
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by the user. The final level of interaction space provides additional hardware 
independence. Components are broken down into user interface primitives 
(UlPrimitive) 418, such as identifiers and selectable spaces to allow the interaction team 
to individually handle information and interactors. This separates, for example, the 
name of a choice from the actual choice, allowing the team to compose both labeled 
buttons ("Print"), where the information appears on the interactor, and telephone 
interactions ("To Print press 1"), where the information about the action must come 
before the name of the interactor. 

The various levels of the interaction class model are compositional. At each 

level, the classes are composed of objects from classes at the next lower level. A user 
interface application is composed of user interface tasks, which are composed of user 
sub tasks, and so on. This compositional pattern provides control over the resolution of 
high level objects such as windows, graphs, diagrams and datasets into lower level 
objects such as scrollbars, timelines, pushbuttons and edit boxes. 

The interaction class model levels are shown with further detail in Figure 5, and 
correspond to the application compositions/windows in the user interaction, the 
interactions contained within each composition, the components used for each 
interaction, the interactors that make up individual components, and the interaction 
primitives that are used by the interactors to realize themselves, All of these classes are 
specializations of an abstract user interface object 510, and thus inherit from the parent 
class the knowledge of how to create, modify and kill themselves as well as several 
variables and a property as shown. The user interface object 510 is the root node for the 
DIG class model. All interaction classes know how to create themselves. If they are 
abstract classes and can't be instantiated until they are concretized, they trap the 
createself operation and send it on to their most appropriate specialization. Concrete 
classes pass the CreateSelf operation through their parent and on to a root node, which 
executes the instantiate of the concrete class by setting its variables comprising 
Referent, Owner, Owns, and UIName, executing any category-specific operations, 
determining the number of sub-components the object has, and sending CreateSelf 
operations down to each of the sub-component classes. Terminal category classes such 
as UlPrimitives 522 execute their own internal operations instead of instantiating 
sub-components. 
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Every obj ect in. the interaction space has a variable Referent which is used to 
identify an object referred to. Usually it is an object in the domain model. Every object 
in the interaction space also has variables, comprising Ovraer and Owns, which simply 
refer to which objects it is contained in or contains. . Variable UIName is simply a name 
5 given the object when it is instantiated. A pseudo code representation of the interaction 
class model is provided in Appendix A. 

Each of the interaction levels shown are themselves representative of full 
specialization hierarchies whose leaf nodes are concrete classes that are instantiated 
into user interface objects during the process of user interface generation or 
10 modification. The levels have the capability to modify parameters, such as the Feature 
parameter, and also may specify further operations, such as SetFileName, 
ResolveChildren and Setlndividual, including the operations inherited from UI Object 
510. The levels comprise a UI Application level 512, UITask level 5 1 6, UlSubtask level 
5 1 8, UlComponent level 520, and a UlPrimitive level 522. Also shown are InFac 3 1 4 
15 which manages the user interface objects 5 1 0 and the JavaTech 3 1 6 which can only see 
■ the interaction space to synthesize the actual interaction with the user. 

As an example, the composition of the UlComponent level 520 is shown in 
simplified block form in Figure ,6 generally at 610. The instantiable classes are the leaf 
nodes at the bottom of the hierarchy. One of the benefits of representing the classes at 
20 each level as a specialization hierarchy is the ability to specify abstract classes, which 
can be resolved at run time based on dynamic situation attributes. Some of the abstract 
classes in this representation include Dynamiclnformation 612 and Staticlnformation 
614. The Dynamiclnformation class 6 1 2 has subclasses comprising Individualltem 6 1 8 
and OfSet 620. Individualltem class 618 has further subclasses Actionltem 624, 
25 Objectltem 626 and Stateltem 628 which are leaf nodes. OfSet 620 has further 

subclasses comprising leaf node NOfSet 632 and OneOfSet 634 which represent the 

cases where the interface is not of a set or is one of a set respectively. 

OneOfSet class 634 is comprised of a Continuous class 636 and a DiscreteSet 

class 638. Continuous class 636 further comprises leaf node RelativeValue 640 and 
30 SpecificValue 642, which comprises leaf nodes SpecificNumberValue 644 and 

SpecificTimeValue 646. DiscreteSet 638 further comprises leaf node ExpandableSet 
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650 and FixedSet class 652. FixedSet class 652 comprises leaf nodes HierarchicalList 
654 and SingleList 656. 

Staticlnformation class 614 comprises leaf nodes Contextual 660, Axis 662 and 
Legend 666. It also comprises an IndividualValue class 664. This class is further 
5 comprised of leaf nodes Single Value 670 and ValueOverTime 672. 

In addition to the hierarchical specialization levels, the classes at the 
UlComponent level are composed of objects from classes at the next lower level and 
also make up object of the next higher level. An application user interaction is 
composed of compositions, which are composed of interactions, and so on. This is the 
10 compositional pattern, which provides more control over the resolution of high level 

objects such as windows, graphs, diagrams and datasets into lower level objects such as 
scrollbars, timelines, pushbuttons and edit boxes. 

An example of one segment of the component hierarchy is shown in Figures 7A 
and 7B generally at 710, which shows a small portion of the UlComponent 520 
15 hierarchy. A first abstract component of UlComponent 520 is a Dynamiclnformation 
component 714. One or more of the Dynamiclnformation components are used to form 
subtasks such as ItemRelationship 716, Discourselnteraction 718, ActionSelection 720 
and ObjectData 722, The. Dynamiclnformation component is specialized into an OfSet 
abstract component 730, which is further specialized into a OneOfSet abstract 
20 component 732. The OneOfSet abstract component 732 is specialized into a 

DiscreteSet component 740 which is used to form a TopicSelection subtask 742, and is 
composed of low level primitives comprising OnOffButton 744, CategoryLabel 746, 
ValueLabel 748 and Grouper 750. Grouper 750 functions to cause a line being shown 
around related primitives, such as a group o OnOffButtons representing multiple 
25 different color selections. 

DiscreteSet 740 is further specialized into either a FixedSet component 760 or 
an ExpandableSet component 762. ExpandableSet component 762 is a leaf node 
composed of a TextEnter primitive 764 which facilitates the entering of a new text value 
such as a new color which is not one of the fixed set of colors. FixedSet component 760 
30 is composed of leaf nodes SingleList 766 and HierarchicalList 768. Finally, an 
ObjectSelection subtask 770 is made up of 1 HierarchicalList 768. 
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As in previous figures, parameters shown in the components are identified with a 
"p" prefix, while the number of primitives are identified by a prefix "a" corresponding 
to one primitive, or "n" for more than one, :Each specialization of components can 
modify the parameters to be representative of the path of specialization. 
5 Two important interaction generation abilities are made possible by the 

organization and contents of this model: 

- The ability to retrieve and enter the model at any node, based on an interaction 
need perceived when there is a change in an application object, and create an 
interface specification down through the compositional hierarchy starting at that 

IQ node. Given the need for a particular interaction, such as a trend display, the 

constituent components, their interactors, and the individual interaction 
primitives are automatically generated and assigned the proper values, labels, 
and relationships. 

- The ability to specify abstract components, and have them resolved based on 
15 the interface context and the state of appHcation objects, such as the type of 

information to be represented and the type of information to be obtained from a 
user. For example, a process object may use different controls (sliders, gauges, 
or pushbuttons) depending on its state and the type of display on which it 
appears. Because of this, the controls associated with a displayed process object 
20 are represented in the compositional model as abstract classes. In the interactor 

model, a component may be defined as composed of a statebutton, which is 
abstract and cannot be instantiated. The statebutton contains information about 
how to resolve itself based on the data requirements of the higher level 
component ofwhich it is part. If the information to be represented is exclusive, 
25 it is resolved as a radiobutton, which must be further resolved based on the type 

of information represented. 

The current interaction model is easily modified as iterative development 
of the interaction generation system continues. However, in its current state as 
described, the organization, compositional outline and the individual properties, 
30 parameters, and methods of the individual classes are sufficient for the dynamic 
generation of process interactions. 
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An example of one segment of the UISubTask component hierarchy is shown in 
Figure 8 generally at 810,'which shows a small portion of the UISubTask 518 hierarchy. 
A first abstract component of UISubTask 51 8 is a Definitionlnteraction 816, which is 
further specialized into an ItemOverTime subtiask. A History Task 840 and PlanTask 

5 842 are both comprised of ItemOverTime subtasks. The ItemOverTime subtask 820 is 
made up of subparts Legend 824, Axes 822 ValueOverTime 828 and SingleValue 826. 

An example of a runtime specialization of a component is shown in Figure 9 A. 
An abstract Dynamiclnformation class 91 0 is matched to various parameters of a data 
object represented at 91 1 . The data object contains many variables having to do with 

10 the color of a thing being represented, such as an airplane in the display of Figure 2. 

Further detail of the choices made at each component in Figure 9A is shown in Figure 

9B, with numbering done consistently with the letter "b" following the number used in h 

9A corresponded to the block at which the choice is made. The path selected is 

indicated by the variable Feature being set to one of the choices indicated at each 

15 component level in Figure 9B. Because the value of the data is one of a set of known 
choices, the DynamicComponent 910 is specialized to an OfSet component 912, 912b. 
Because the choice is exclusive, it is specialized to a OneOfSet component 914, 914b 
and so on through a DiscreteSet component 916, 916b, FixedSet 920, 920b, and 
SingleList 922, 922b until, a task in the ISpace is instantiated as xSingleList:Golor as 

20 indicated at 924. Eventually, this may be realized on a display screen through the 

Techs/modules as a set of radio buttons or a drop down list on a display screen or as 
verbally presented menus on a telephone presenting the currently selected value of the 
data object. 

DiscreteSet component 916 contains n OnOffButton primitives, which is 
25 determined with reference to the data object 91 1 wherein the number of choices is seen 
as three, Red, Yellow and Blue. Perhaps better color selections are available, but the 
choices are more than adequate for use as an example. Further references to the data 
object utilized in the specialization of components are indicated by the lines shown 
between the components and the variables in the application object 911. 
30 The model also provides the ability to enter the model at any node, as described 

above, based on the interaction need, and create an interface specification dovm through 
the compositional hierarchy starting at that node. For example, given a need for a 
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particular Ullnteraction, such as a trend display, the constituent UlComponents, and the 
individual UI Primitives are automatically generated and assigned the proper values, 
labels and relationships without further: intervention. 

This instantiation cascades through the elements that are defined as the 
5 compositions of each class, and the required model elements are instantiated, referenced 
and specialized: Each class and object knows how it is composed. Abstract elements 
are resolved where necessary by matching properties of more specific elements with 
properties of the data objects they will represent. At each step, the currently instantiated 
object investigates its owners and context, and sets its parameters, including its 
10 relationships to real world process objects according to the situation it is in. 

The technician, first referred to at 3 1 6 and 3 1 8 is a code module responsible for 
realizing interactions on an interaction device set, which consists of one or more input 
and output devices such as microphone/speakers, keyboard/monitor, etc. The technician 
or technicians know how to convert instantiated, concrete interaction objects in the 
15 interaction space into the user interface toolkit suitable for its associated device set. For 
example, JavaTech 316 converts a single list element into a group of radio buttons while 
the SpeechTech 3 1 8 will convert it into a spoken telephony menu. 

Techs can also interpret human interactions with the presented user interface 
objects into changes to the interaction space and report them back to the InFac 3 14. 
20 When the user interacts with the user interface presentation, the interaction space objects 
know what actions to take, such as invoke a display need like "Show my controls'' or 
"Graph my past trend values", or an application need, like "Turn on this fan" in the case 
of a building management application. These requests are generated by the user 
interface objects when the user interacts wdth the system. They are then interpreted as 
25 interaction space changes by the tech, and acted on by InFac. If the need is strictly a 
display need, it is handled within the Interaction Management Team through 
modifications to the interaction space. If it is an application need, it is relayed to 
external agents that can meet the need. 

Within a multi-agent system, the interaction management team, specifically 
30 InFac, generates interaction needs and user information needs that are fulfilled by other 
agents, and itself fulfills needs from external agents. The interaction needs are 
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determined by the user's current context and tasks, and as a reaction to the requests 
received from other agents. " • . . 

For example, suppose the user wishes to monitor a thermostat's set-point and the 
measured room temperature over time, and indicates this by selecting a button displayed 
5 • by the JavaTech. The button rendered by the JavaTech corresponds to an instantiated 
Objectltem representing the thermostat within the interaction space. 

The Tech informs InFac of the user's action in the context of this class, and 
InFac responds by managing the content and structure of the current interaction space. 
If follows estabUshed best practices process heuristics to select a required task or 
10 interaction within a task, and requests the appropriate interaction element to instantiate 
itself as a representation of a particular process object. 

At this point, without knowing the specific data to be displayed, InFac can only 
resolve the interaction object to an abstract class within the interaction space. InFac 
also determines that the infomiation needed to maintain this .on-going interaction - the 
15 temperature and setpoint - are provided by an external agent. On receiving the- data, 

InFac can resolve the abstract object into a concrete one by matching the properties of 
the data with the features of interaction elements. In the case of the thermostat, the 
multiple instance of time-value data which represent past static value information are 
determined to require a ValueOverTime trend display. 
20 This results in a fully specified interaction-space model, where each level knows 

the information relevant to create itself and to realize the interaction. For example, a 
pushbutton knows what action request to send back to inFac and how to label itself A 
menu knows how to order itself, and a navigation frame knows what application is 
being presented for navigation. In the thermostat example, a ValueOverTime trend 
25 display knows what data to display, how to label its axes, and so on. In this way, InFac 
causes changes to interaction space for an on-going application through creation or 
modification of user interface elements at various levels. 

Finally, InFac sends a message to the JavaTech to realize the ValueOverTime 
object. The JavaTech uses details of the instantiated object to determine specifics of the 
30 trend graph to display the minimum and maximum bounds for the temperature and the 
length of the time axis, for example. The JavaTech also begins monitoring the 
interaction space object for changes. For example, as the temperature or setpoint 
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changes as time passes, InFac updates the interaction space object to reflect this, which 
in turn, causes the JavaTech to update its display, of the trend. 

A class model diagrain of interaction management authorities is shown in block 
form in Figure 1 0 generally at 1 0 1 0; A rbot node for software modules is shown as 
5 represented at 1 0 1 2. A prototype parent class indicated as Society Agent 1 0 1 4 has InFac 
3 14 as a specialization and also a member of an Interaction Facilitator class 1018. 
Similarly, a Technician 1016 parent class has a subclass JavaTech 1 0 1 9 of which 
JavaTech 316 is a member. Broken lines are now used to indicate the relationships of 
these classes to other objects. The highest level of application object is represented at 
10 1020 as APPOBJ having several variables and primitives. The interaction space is 

represented at 510 and labeled UIOBJ, and a UIDevice is indicated at 1040. Further, a 
box labeled Heuristics 1 030 comprises a lookup table containing knowledge of device 
capabilitites indicating how to map elements to the device. 

■ Instances of the InteractionFacilitator class 1018 may only read information 
15 from APPOBJ 1 020, while having the ability to manager the interaction space 510. 
Instances of the JavaTech class 1019 again may only see APPOBJ 1 020 in order to 
obtain needed information. This ensures that the interface designer cannot change the 
application, but only react to' it. The JavaTech 1019 instances can only see the 
interaction space 5 10, but uses the UIDeVice 1 040, providing it the information and 
20 controls necessary to properly implement the dynamically designed interface utilizing 
the heuristics 1030. 

An application object domain model is shown in Figure 11 . A generic 
application object is shown at 1 11 0 with variables and primitives. Two specializations 
of the generic APPOBJ object 1110 comprise a DomainComponent 1 11 8 and a 
25 DomainPrimitive 1 1 14. Each DomainComponent 1 118 is composed of one or more 
APPOBJ objects. DomainPrimitive 1 1 14 is specialized into UniqueDataltem 1 120 or 
ConstrainedDataltem 1 142. UniqueDataltem 1 120 is instantiable into leaf nodes 
SingleAction 1 1 22, UniqueName 1 1 24, UniqueNumber 1 1 26, GenerallD 1128, 
DescProse 1 1 30 or SingleEntity 1 140. ConstrainedDataltem 1 142 is further 
30 specializable into TimeDataltem 1 1 44, ValueDataltem 1 1 46 or DiscreteDataltem 1148. 
TimeDataltem 1 144 is instantiable into leaf node DateTimeGroup 1 152. 
ValueDataltem 1 146 is instantiable into leaf node Location 1 154 or may be specialized 
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into ContinuousDataltem 1 150 which is instantiable into leaf nodes Duration 1 160, 
Speed 1 162, Temperature 1.164 and Quantity 1 166. Leaf nodes Duration 1 160, Speed 
11 62, and Temperature 1 1 64 each have a variable for Units. ContinuousDataltem 1 1 50 
also has several variables listed, including the Units variable which is defaulted to Null. 
The other variables include mininium and maximums, alarm and warning limits and 
other variables which are declared to be real, integer or boolean as shown. 

DiscreteDataltem 1 148 is further instantiable into leaf nodes Qualit>' 1 170, 
Property 1 172, State 1 174, Entity 1 176, Action 1 178, Objective 1 180, and Cause 1 182. 
It is also specializable into a TimeSet 1 184 with leaf node MonthValue 1 1 86. Each of 
the leaf nodes described provide information or allovy for an interaction with a user. 

There are also several leaf nodes representing files of information used by the 
Domain model. These comprise ContextRepresentation 1 188, GraphicRepresentation 
1 189, Schedule 1 190 and ValueHistory 1191, each of which are leaf nodes of an 
abstract class DATAVIEW 1 192 having a parameter Filename indicating the address of 
i the files. ContextRepresetnation 1 1 88 and GraphicRepresentation 1 1 89 are referenced 
by the generic application object, and hence available to all objects in the model. The 
Schedule 1 1 90 and ValueHistory 1191 leaf nodes are referenced by the 
DomainPrimitive class, and are available to it and classes of \yhich it is comprised. 

One further example of the generation of an interaction space is shown in Figure 
20 12. This is an example of the generation of a search and rescue report, as shown in 
Figure 2. Only the first three levels of the interaction space are shown, the 
UIApplication level 410, the UITask level 412 and the Ullnteraction level 414. The 
search and rescue application is represented by the block labeled ObjectApplication at 
415 which is a specialization of UIApplication 410. The current mission is indicated at 
25 416 as a specialization of ObjectApplication 415. ObjectApplication 415 is composed 
of a DefmeTask object 1220 which is a specialization of a ViewObjectTask 1224, which 
is further specialized from the UITask class 412. 

DefmeTask 1220 is further composed of a an ObjectData subtask 1230 which 
itself is composed of multiple DynamicComponents 910. ViewObjectTask 1244 is 
30 fiirther composed of an ActionSelection subtask 1 240, which itself is also composed of 
multiple DynamicComponents 910. The ActionSelection subtask 1240 is a 



wo 99/34284 PCT/US98/25968 

' -20- 

specialization of a Selectionlnteraction subtask 1242, which is specialized from 
SubTask-414. ^ ■ ^ • ^ ^ 

The report may be'cbmpleted by the agents assisting the search and rescue 
process, the form must be reviewed arid authorized by a human user before being 
5 submitted. When authorization of the report is required, a display need is generated for 
the form: InFac attempts to meet the need by notifying the user that it would like to 
interrupt the current task if necessary and present the report approval. When the user 
concurs, InFac determines that the search and rescue information is categorized as of the 
definition type, and begins instantiating a DefmeTask object 1220 in the interaction 

10 space. The search and rescue data is then used to specialize each level of the hierarchy 
as described with reference to Figure 9. 

The Define Task is made up of a single ObjectData Interaction 1230, After 
setting its own properties such as its owner, etc., the newly instantiated search and 
rescue Define Task sends a message to the ObjectData Interaction Class to create itself, 

15 resulting in a search and rescue ObjectData Interaction. 

Since the specific content of the search and rescue form cannot be determined in 
advance, the Data Interaction objects need to resolve their compositional structure to 
determine how many individual Components they need. For example, information 
fields pertaining to aircraft data are not relevant if the situation involves a submarine. 

20 This will be reflected in the search and rescue data and, consequently, can only be 
determined based on the specific situation. 

Once the number of components are determined, the search and rescue Data 
Interaction object creates the Dynamic Component class 1236 for each required field. 
Because a Dynamic Component is an abstract class, details of the search and rescue 

25 fields are matched to the defining properties for dynamic components to resolve it into 
concrete classes. A process analogous to that takes place for each search and rescue 
field to create the required components in the Interaction Space. 

In one example, the aircraft color field 222, which is currently set as "Orange", 
but has allowable values "Blue | Orange | Red | Green" may be part of the form. Based 

30 on the defining parameters of the data, the Dynamic Component is resolved into a 
SingleList Component (see Figure 9), which is composed of four OnOffButton 
Primitives (one for each allowable choice), with the one representing "Orange" having a 
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selected state and the others having an unselected state^, four ValueLabel primities (one 
to label each allowable choice), a CategoryLabel Primitive for the field label "Aircraft 
Color", and a Grouper Prirnitive tp .bpund the component. The Information Label of the 
SingleList Component is not ^required, .since the Aircraft Color property has no 
engineering units. Each of these Primitives are created as they are determined. , 
When the interaction space is completed for this display need through the 
process described above, the Interaction Facilitator directs the Tech to generate the 
search and rescue situation report for the user who must review it. The Tech will 
generate the report based on the user, the available hardware and the Tech's own 
knowledge. For example, the report might be presented as a displayed form on a screen, 
or a faxed hardcopy might be sent to a remote user via available means. 

As the application of the DIG process to search and rescue shows, it is possible 
to use dynamically changing task and display requirement to compose on-line designs 
of user interactions. Because it is generated in direct response to the properties of the 
data and the user, a DIG based interaction is inherently dependent on the situation, 
centered on the user and the task, and preferably firmly rooted in the principles of good 
user interface design. 

DIG benefits both developers and user of mixed initiative assistive systems. 
Developers are assured good design and a consistent look.and feel to applications. 
Products can rapidly and easily modify and create user interfaces to their system 
through modification of the DIG interaction models. Reuse and automatic 
documentation capabilities help reduce development costs and effort. Incremental 
advances in computing capabilities, languages and infi-astructures can quickly be used to 
enhance products. 

A further advantage of the invention is that users are not required to perform 
laborious interface definition and configuration. They are automatically provided with 
well designed state of the art user interfaces. They also feel more in control and 
connected to their tasks with a consistent interface over varying situations. Upgrades 
and model evolution are provided with dynamic responsiveness to context, task, 
situation and user intentions without the need for retraining and support. 

It is to be understood that the above description is intended to be illustrative, and 
not restrictive. Services may be implemented in hardware, ASICs or software or any 
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combination thereof. When the services and methods are implemented in software, they 
may be stored in any available computer readable medium as represented at 130, and 
executed normally from a randorn access memory associated directly with a processor. 
Such computer readable medium includes disk dives, floppy drives, CD ROM, tape, and 

5 even includes remote storage devices siich as that accessible via network. Further object 
oriented models which basically encapsulate data and progranmiing into identifiable 
files may also be utilized. The invention was described with respect to a model having 
five levels. It is easily contemplated that models having a different number of levels 
may be used without departing from the scope of the invention. While many of the 

10 embodiments have referred to a user as a person, it is clear from the description that a 
user may also be another program or object. In that case, the interfaces used may be 
program to program interfaces. Such interfaces, when synthesized dynamically based 
on needs may help conserve system resources such as main memory, which will help 
applications run. faster and more efficiently. Many other embodiments will be apparent 

15 to those of skill in the art upon reviewing the above description. The scope of the 

invention should, therefore, be determined with reference to the appended claims, along 
with the full scope of equivalents to which such claims are entitled. 
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CATEGORY STRUCTURE 



SARDIG 

UlApplications 

urrasks- 

UISubTasics 
UTComponenis 
UlPrimitives 
InteractionClasses 

VarTypes 

DomainClasscs 

[nteraciionManagemcnt . 



INTERACTION MODEL 



The Interaction Classes category holds the root class in the DIG modeU UIOBJ, All objects in an 
[nteraction-Space are specializations of UIOBJ. 

UIOBJ 

This is (he root node for the DiC class model. All Interaction Classes know how to Create 
themselves: if they are abstract classes and can't be instantiated until they are concretized, they 
trap the CreateSelf operanon and send it on to their most appropriate specialization. Concrete 
classes pass the CreateSelf through their UI Category parent and on to this root node, who 
executes the instantiate of the concrete class by setting it's v Referent. vOwner and vUIName, 
executing any category- specific operations, determining the number of sub-components the 
object has, setting its vOwns, and sending CreateSelf operations down to each of the sub- 
component classes. Terminal category classes (UlPrimitives) execute their own internal 
operations instead of instantiating sub-components. 

Public Properties: 

V Referent : OBJ 

Every object in the Interaction- Space has a vRefercni of lype 
REFERENT which is ihc object in the rcai world the [SObjeci •refers * 
lo. Usually it is an object in the Domain Model. The constructor 
funcuon is passed the referent as the second argument. 

vOwner : UIOBJ 

Every object in the Interaction-Space has a vOwner. The vQwner of the 
root node of the space (the UlAppUcation object) is owned by itself. 
Except for the root node, all objects are owned by the next higher 
category; e.g., applications own tasks; components own primitives. 
The constructor function is passed the name of the owner as the third 
argument. 

vUwns : LIST(UIOBJ) 

Every object in the Interaction- Space except UlPrimiuve terminal nodes 
have a list of UlObjects that are contained wiihin themselves. This list 
IS set after the set of children is resolved at the category level, and 
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LOGICAL VIEW REPORT 

synchronized after ail children arc instantiated to accommodate 
ipeciaiization thai occurs alter the initial number of children is 
determined. 

vUIName : STRING 

Every object in the Interaction-Space is given a UlName when it is 
instantiated. This is passed in the constructor function as the first 
argument of CreaicSeif, 

pFeaturc : FEATURE 

Every class in the UIOBJ hierarchy has a pFcaiure attribute which 
distinguishes it from siblings at the same level which are 
specializations of the same parent, to allow classes have unspecialized 
sub-classes until instantiation. For example, the pFcature of a 
OneOfSci UlComponent that distinguishes it from an NOfSei 
UlComponeni is the feature +exclusive. 

Public Methods: 

CreatcSelf (UlName : STRING = default. Referent : REFERENT = default, 
UlParcnt : InteractionClassRoot default) : 

The CreatcSelf of a concrete. UlObject is not trapped by the object or its category, but is 
perpetuated up to the root node. . 

CreateSelf (UlName.' Referent. UlParent) 

set my pUlNamc 

set my pReferent 

set my pOwner 

perform category operations 

set initial pQwns 

send CreateSelf to each of my children (pOwns) 
, update pOwnis for specializations 
end CreatcSelf .\ . V * . 

lIIAppjicsTtToiSii^^ss^w^^ • ' • 

The U [Applications category is the highest level category. represerMing an applicatiorr as a set of related 
tasks performed on the same data. Ul Applications are composed of UlTasks. 

^^%^IApific^.]Tis the root node of. he application category. It holds an application tfrorntHe 
userspo.nt ofvie^): a coherent grouping of related tasks on a set of data that the userj,^s 
OS related and interdependent. A set of tasks required to manage a search and rescue operatwn. 
or to tnanage a building, or to provide point-of- request information systems, for example. 
Derived trom UIOBJ 
Public Properties: 

vDataFileName : FILENAME ^ ^ 

vDataFileName is set when the application is first instantiated, and 
refers the the pathname of the domain data records for ihe application. 
When referring to the data of their actual referents, individual objects 
will append the referent name to this filename. 
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LOGICAL VIEW REPORT 



pFeature : FEATURE = ^application 

Public Methods:, , . .: : > ^ 

SetFiIeName () : ; , 

SeiFileName sets ihe path for the entire application's data. Hard code for SAR: for 
example: <appname>.dbs. <appname>.grc. The fiienanne is used by the individual objects 
to-set their referents, and in future to get additional data like repon formats and such. 

SARApplication 

A speciaiization of the Uf Application --> Object Application classes for the joint forces search 

and rescue domain. 

Derived from ObjectApplicaiion 

Object Application 

Object applications are specializations of U I Applications that have the task set required to 
support a dynamic object base that represents a dynamic process that can be controlled ( rather 
than just observed). There are two conversation tasks: user and system dialogs: four global 
(application) tasks: initiate a conversation, overview the system, select a domain object, select 
a view of a domain object: four object specific tasks: define object, view object history, view 
object plan, view object's sub-components and/or super-components. 
Derived from UIApplicaiion 
HasA Constituents: 

aSystemOvervicw : OverviewSystem 
aAppObjectList : SelectAppObject 
aViewList : SelectView 
aDefineTask : DefineTask 
aHistoryTask : HistoryTask 
aPlanTask : PlanTask 
aRelationshipTask : ReiationshipTask 
aUserDialog : UserlnitiatedDiaiog 
aSystemDialog : SystemlnitiatedDialog 
aDiaiogManager : Cbnversationlnitiation 

The UlTasks category contains the specialization hierarchy for the task level of the Interaction- Space. 
UlTasks are akin to things like plan and monitor UlTasks are composed of UlSubTasks, and 
themselves compose U! Applications. 

UITask 

UTTasks are the main pieces of an application. They correspond to different views of data and 
global interaction management functions: present values (define), present context 
(relationship), past values (history), select view, initiate conversation, etc. 
Derived from UIOBJ 
Public Properties: 

pFeature : FEATURE = +task 

The pFeature is an attribute of all classes of UIOBJ. 

pContext : CONTEXT 

The pContext of a UITask is the type of object the referent of the task 
is: a SarApplication or a SARObject. 
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L O G I C A L V I E W R E P O R T 



pFormat : FORMAT " ' 

The pFormat of a UlTask is the general format of ihe task in the 
prototypical case. The pFormai for a global task is Choices, for a 
conversation task a dialog. 

pTimeSpan : . TIMESPAN 

Tfftf" pTimeSpan of a UITask is the timespan in which the task data 
occur, relative to now: Past, Current. Future. 



GlobalTask 

Global tasks manage the v/ew object tasks globally^ by presenting sets of process space 
objects and process space views, and an overview of the entire process space. 
Derived frnm UlTask 
Public Prcipeaies: 

pFeature : FEATURE = ^application 
pContext : CONTEXT = APP 
pFormat : FORMAT = Choices 
pTimeSpan : TIMESPAN = Current 

OverviewSystem 

Overview system provides a view of the entire set of process space objects. 
DerivedfromGlobalTa.sk 

Public Propenies: , . , 

pfcature : FEATURE = -»-status 
HasA Constituents: 
aObjeciData : ObjectData 

Select AppObject , 

Select Application Object task allows selection of a specfic object in the process space. 

Derived from GlobalTask 
Public Properties: 

pFeature : FEATURE = +-objects 
HasA Constituents: 

aObjectSelection : ObjectSejcction 
SelectView 

Select view tasks provide selection of a specific view available for a specific object in the 

process space. 

Derived from GlobalTask 

Public Properties: 

pFeature : FEATURE = +views 
MasA Constituents: 
aViewInvoker : InvokeView 

Conversationlnitiation 

Manage dialog tasks allow user to initiate a user initiated or other kind of dialog. 
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Derived from GlobalTask 

Public Properties: , 
pFeature : FEATURE = -t-discourse - 

HasA Constituents: , - : \ • 

aToptcSelection ; TopicSelection 
~« • 

ViewObjectTask 

View object tasks involve taking a particular point of view towards a specific object in the 
process space. 

Derived from UITask ' - 

Public Properties: 

pFeature : FEATURE = ^-object 
pContext : CONTEXT = OBJ 
pFormat : FORMAT = Frame 

HasA Constituents: 

aObjectToViewSelection : ObjectSelection 
aTaskActionSelection : ActionSelection 
aViewDismisscr ^: DismissView 

PlanTask 

Plan tasks view objects from the perspective of future behavior. 
Derived from ViewObjectTask 
Public Properties: 

pFeature : FEATURE = +plan 
pFormaC : FORMAT = Trend 
pTimeSpan : TIMESPAN = Future 

HasA Constituents: 

altemOverTime : ItemOvcrTimc 
aTimeSelection : TImeSelcction 



HistoryTask 

History tasks view an object as a set of past time stamped data. 
Derived from ViewObjectTask 
Public Properties: 

pFeature : FEATURE = ^review 
pFormat : FORMAT = Trend 
pTimeSpan : TIMESPAN = Past 

HasA Constituents: 

altemOverTime : ItemOvcrTimc 
aTimeSelection : TimeSelection 

DefineTask 

Define tasks view an object as a set of components and values. 
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Derived from ViewObjectTask 
Public Properties: 

pFeaiure : FEATURE = +derinition 
pFormat : FORMAT = Form 
pTimeSpan : TIMESPAN = Current 
HasA'tonsiiiuents: 
aObjectData : ObjectData 

RelationshipTask 

Relationship tasks take the point of view of the subcomponents of an object and their 
relationships to each other (a 2D map, generally) 
Derived from ViewObjectTask 
Public Properties: 

pFealure : FEATURE = ^context 
pFormat : FORMAT = Map 
pTimeSpan : TIMESPAN = Current . 
HasA Cunstituents: " 
aitemRelationship : ItemRelationship 

ConversationTask 

Conversation tasks support discourse between human and computer collaborators. 
Derived trom UITask 
Public Properties: 

pFeature : FEATURE = +meta 
pContext : CONTEXT = APP 
pFormat : FORMAT = Dialog 
pTimeSpan : TIMESPAN = Current 

HasA Consiituenis: 

aGlobalActionSelection : ActionSelection 

SystemlnitiatedDialog 

A conversation task that is initiated by the system. 
Derived from ConversationTask 
Public Properties: 

pFeature : FEATURE = +systemtouser 

HasA Constituents: 

aDiscourse : Discourselnteraction 



UserlnitiatedDialog ^ . . 

User dialogs are dialogs that are initiated by the user through the Conversation! ntiation Global 
task. 

Denved from ConversationTask 
Public Properties: 

pFeature : FEATURE = -t-usertosystem 

HasA Constituents: 

aDiscourse : Discourselnteraction 
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UISubTasks^ 

The UlSubTasks category is the specialization hierarchy for the basic interactions in a Ul They are 
akin to the level of trend graphs, schematis. and. task bars tn a graphical ill. UlSubTasks are made up 
of UiComponents and themselves make up U [Tasks. 



UISubTask 

UlSubTask is the root node of the SubTask category in UIOBJ. It contains the subtasks 
within the tasks within the application, and is composed of UiComponents. 
Derived from UIOBJ 
Public Propenies: 

pContext : CONTEXT = OBJ 

pContext is a fixed attribute of all UlSubTasks. The pContexi of a 
UISubTask is the type of domain object or domain subobject that can 
be referenced by this subtask. For example, the pContext of an 
ItemOverTime is the value parameter of an application object; for a 
definition interaction it is the subcomponents of an application object. 

pFeature : FEATURE = +subtask 

Public Methods: 

ResolveChildrcn () : 

n = number of SARPrimitives contained in the SARComponeni designated by the 
pReferent of this Interaction. It is resolved as the number of Domain Pnmitives owned by 
this Domain Component that have the same feature as the pFeature of the task; for 
example, the SARComponent "SARIRForm" has x SARPrimitives as children. The 
SARPnmitives that represent the sub-components of the form are marked +data and so are 
shown in the ObjectData Interaction. 

Discourselnteraction 

Discourse interactions between the user and the system. 
Derived from UISubTask 
Public Properties: 

pFeature : FEATURE = ^communication 

KasA Constituents: 

nDiscourseComponcnt : Dynamiclnformation 
Contextlnteractton 

The context mteraction class shows the relationships of an application object. 
Derived from UISubTask 
Public Propenies: 

pFeature : FEATURE = +relationship 
HasA Constituents: 
aContext : Contextual 



ItemRelationship u^^frh. 

The Item Relationship class shows the subobjects of an appobject; the relationship of the 



item to it's parts. 
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Derived from Coniexilnieraction 
Public Properties: * * \ . • 

pFeature : FEATURE = i-subobject 
HasA Consiiiuencs: 

nReiationshipComponents Dynamic Information 
- » ^ 

Definitionlnteraction 

The definition interaction shows the data for an application object. The only data currently 
defined is the data for the object's sub-objects (the ObjectData concrete class), which is akin to 
■ a form with fields labelled with each of the sub-objects and containing the value of the sub- 
object. 

Derived from UlSubTask 

Public Properties: ^ 
pFeature : FEATURE = +item 

ObjectData 

Object data interactions are definition interactions that show the values of an application 

object's subobjects. 

Derived from Definitionlnteraction 

Public Properties: 

pFeature : FEATURE = +vaiue 

HasA Constituents: 

nOataComponent : Dynamiclnformation 
aCurrentVaiue : SingleValue 

ItemOverTime 

The item over time shows the relationship of the domain ' object 's parameeter valueis) to past 
or future time. 

Derived from Definitionlnteraction 
Public Properties: 

pFeature : FEATURE = +overtime 
HasA Constituents: 
aLegend : Legend 
aAxesSet ; Axes 
aCurrentValuc : SingieVaiue 



Selectionlnteraction 

A selection interaction is in the context of an application object where the feature ts 
type of choice. There are four types currently defined: action choice, object choice, u 
choice, topic choice. 
Derived from UISubTask 
Public Properties: 

pFeature : FEATURE = ^choice 



ActionSeiection 

The action selection subtask provides the choice of an action within the context of 
application object 
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Denved from Selectionlnteraciion 
Public Propenies: 

pFeature : FEATURE = +action . 

HasA Constituents: 

nActionComponent : Dynamiclnformation 
~ « _ 

ObjectSelection 

The object selection subtask provides for the selection of an object from a list of objects that 
are related to one another. 
Derived from Selectionlnteraciion 
Public Properties: 

pFeature : FEATURE = +obiect 

HasA Cunstiiucnts: 

aObjectHierarchyList : HierarchicalList 



TimeSelection 

The time selection subtask provides the capability to select a time period within which to 

view information, 

Denved from Selectionlnteraction 

Public Properties: 

pFeature : FEATURE = -^time 

HasA Constituents: 

aTimeValueSelector : SpecificTimeValue 



TopicSelection 

The topic selection allows the choice of all the domain primitives defined for the current 
object that are coded as -atopic 
Denved from Sclcctionlnicraciion 
Public Properties: 

pFeature : FEATURE = +topic 
HasA Constituents: 
aTopicList : DiscreteSet 

View Interaction 

The view interaction class provides a way to invoke and dismiss view tasks as component 
parts of the view; the specializations are invoke the view and dismiss the view. 
Denved from Selectionlnteraciion 
Public Properties: 

pFeature : FEATURE = +view 
HasA Consiituenis: 
nViewAction : Actionltem 



InvokeVlew 

Provides a method to invoke a view on an object. 
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Derived from Vtewlnieraciion 
Public Properties: 

pFeature : FEATFURE = +invoke 
Dismis«yiew 

Provides a way to dismiss a view of an object. 
Derived from Viewlnteraction 
Public Properties: 

pFeature : FEATURE = +dismiss 



UlComponents 

The UlComponents category contains the specialization hierarchy for the component level, akin to the 
widget level in a graphical user interface. UlComponents are composed of UlPrimitives and themselves 
compose UlSubTasks. 



UlComponent 

A UlComponent is the root node of the UlComponent category. UlComponents appear within 
UlSubTasks and are composed of UlPrimitives. 
Derived from UTOBJ 
Public Properties: 

pFeature : FEATURE = ^component 

The pFeature of a UlComponent is the feature that is used to specify an 
abstract component into a more concrete component. 

pResult : RESULT = NULL 

The pResult of a UlComponent is the result of invoking thai 
component in the UI context. 

Public Methods: 

ResolveChildren () : 

UlComponents resolve the number of children for three classes: NOfSet, Discrete, and 
LargeFixedSel. For Separator primitives, there is one separator for each subgroup of 
objects; for other pmmitives, n = the number of items in the set. 

Dynamiclnformation 

,A dynamic information component is a Ul Component that represents data, rather than 
context, and is therefore potential changable in some way or another 
Derived from UlComponent 
Public Properties: 

pFeature : FEATURE = +dynamic 
pResult : RESULT 

Public Methods: 

CreateSelf () : r . ^ 

[f exisis(vReferent : vChoices) then feature = -^set -> OfSet; else -set -> [ndividualltem 

Individuailtem 

An individual item is an action or an object. 
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Derived from Dynamic In tbmiat ion 

Public Properties: 

pFeature : FEATURE = -set 

Public Methods: 

CreateSeif () : 

irj5Referent: ClassType contains "Action", then feature = taction -> Actionltem; else = 

+object "> Objectltem ^ 
HasA Cimsiituenis: 
altemName : ValueLabel 

altemType : Category Label - ; . 

Stateltem 

Derived from [ndividualhem 
Public Properties: 

pFeature : FEATURE = +state 
pResult : RESULT = ChangeState 
HasA Constituents: 
aStateSelector : OnOffButton 

Objectltem 

Object Items are used for individual object selection (a Fan). The select object behavior of an 
Objectltem is to show the associated controls when on and hide when off. 
Dcnved from Individuallicm 
Public Propenies: 

pFeature : FEATURE = -Kobject 
pResult : RESULT = SelcctObject 
HasA Constituents: 
aObjectButton : OnOffButton 

Actionltem 

Action Items are used for individual actions (Print, Acknowledge) 
Denved from Individualltem 
Public Propenies: 

pFeature : FEATURE = +action 
pResult : RESULT = [nvokeAction 
HasA Constituents: 
altemSelector : SelectorButton 

OfSet 

The OfSet classes are interactions where one or more choices is made from a set of choices: 
- e.g., times, dates, extendable or fixed lists, bounded values. 
Denved from Dynamic Information 
Public Properties: 
pFeature : FEATURE = +set 
Public Methods: 
CreateSeif () : 

if vNumChoices of my vReferent > 1. then -exclusive --> NOfSet: else ^exclusive -> 
OneOfSet 
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NOfSet 

List where more than one item can be selected. " * 
Derived from OtSet 
Public Properties: 

pFeaYure : FEATURE = -exclusive 
pResult : RESULT = Selectltcm 
HasA Constituents: 
aListTitle : CategoryLabel 
nValueLabeis : ValueLabel 
nOnOffButtons : OnOffButton 
aListCrouper : Grouper 
nListSeparator : Separator 

OneOfSet 

Only one item may be selected from a set of items. 
Derived from OtSet 
Public Properties: 

pFeature : FEATURE = +exciusive 

Public Methods: 
CreateSclf 0 : 

If exists(vReferenl : vNumChoices ) then -Kiiscrete*--> Discrete; else -discreie -> 
Continuous 

Continuous 

A continuous value is a numeric continuous value, usually bounded, ' 
Denved frooi OneOfSet 
Public Properties: 

pFeature : FEATURE = discrete 
pResult : RESULT = ChangeValue 

Public Methods: 
CreateSelf 0 : 

if vReferent : vSpecificity = TRUE then +speciric ->SpecificValue; else -specific - 

>RelativeVaIue 
HusA Consiilucnts: 
uValueLabel : ValueLabel 
aUnitsLabel : InfoLabel 
aCategoryLabel : CategoryLabel 
aUpSpin : SpinnerButton 
aDownSptn : SpinnerButton 
aVaiucEnter : TextEnter 

RelativeValue .-r l 

.4 Relative Value provide the entry of a continuous value on a relative scale; 7 more than 
current". " 10 mins from now", "hot". 
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Derived from Continuous 
Public Properties: 

pFeature : FEATURE = -specific ; 
HasA Constituents: 
aValueLocaCor : LocatorButton 
aValueAxis : Axis 

SpeciflcValue 

A specific value is stated as an absolute (7 degrees, 12) 
Derived from Continuous 
Public Properties: 

pFeature : FEATURE = +sp€ciric 

Public Methods: 
CreateSelf () : 

if vReferent isa TimeDataltem then +time -> SpecificTimeValue; else if vReferent isa 
VaiueDataitem then ^number SpectficNumberValue 

SpecificNumberValue 

A specific general number. 
Derived from Specific Value 
Public Properties: 

pFeature : FEATURE = +numbcr 
SpecificTimeValue 

A specific lime value is a specific value that is expressed as a time. 
Derived from SpecificVaiue 

Public Propenies: - ; . , . . 

pFeature : FEATURE = +time 

DtscreteSet 

A Discrete is a discrete set of items. 
Denved from OneOfSel 
Public Properties: 

pFeature : FEATURE = ^discrete 
pResult : RESULT = ExIusiveSelectltem 

Public Methods: 
CreateSelf () : 

if vReferent : vExiendable = TRUE then -fixed --> ExpandableSet; else +tixed --> 

FixcdSct 
HasA Constituents: 
nOnOffButton : OnOffButton 
aCategoryLabei : CategoryLabel 
nDiscreteVAlues : ValueLabel 
aSetGrouper : Grouper 

ExpandableSet ^ . 

An expandable set is a list of discrete items which can have new items added to tr. 
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Derived from DiscreteSet 
Public Properties: 

pFeature : FEATURE = -fixed - 

HasA Constiiuenis: 
aNewChoice : TextEnter 

FixedSet 

A FixedSet is a list of discrete items which cannot be user-expanded. 
Derived from DiscreieSet 
Public Properties: 

pFeature : FEATURE = +rixed 

Public MclhiKJs: 
CrealeSelf () : 

if the items in vReferent : vChoices contain hierarchy delimiters, then +hierarchicai --> 
HierarchicalLisi; else -hierarchical --> SingleList 

HierarchicalList 

A hierarchical list has items organized hierarchically. . ■ . 

Derived from FixedSet ' 
Public Propenies: 

pFeature : FEATURE = ^hierarchical 
SingleList 

A single list has items arranged at a single level. 
Derived from FixedSei 
Public Properties: 

pFeature : FEATURE = -hierarchical 
Staticlnformation 

Information that provides context or translation. 

Derived from UTComponeni • 
Public Propenies: 

pFeature : FEATURE = -dynamic 
Axes 

Axes provide context between a continuous or stepped data value and time. 
Derived from Staticlnformation 
Public Properties: 

pFeature : FEATURE = +time 

HasA Constituents: 
aTimeAxis : Axis 
aDataAxis : Axis 

Legend 

A legend provides equivalence between one representation of an object and another 
representation. 
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Derived from Staticlntormaiion 
Public Properties: 

pFeature : FEATURE = tequivalence 
HasA Constituents: 
aLegendBox : Grouper 
aCategoryLabel : Category Label 
nReferencedTypes : /ValueLabel 
nTranslatedXypes : ValueLabel . 

Contextual 

Context provides a context for information: usually a static graphic. 
Denved from Staticlnformauon 

Public Properties: , , ■ 

pFeature : FEATURE = +contcxt 

HasA Constituents: 
aContextView : ViewLayout 

IndividualValue 

An individual value is either a single value or a set of timed values. 
Derived from Staiiclnformation 
Public Propenics: 

pFeature : FEATURE = -t-value 

HasA Constituents: 
aLabel : Category Label 

ValueOverTime 

A ValueOverTime is a set of values for a single parameter that happen over rime: a set of 

time-value pairs. 

Derived from [ndividuaiValuc 

Public Properties: - 

pFeature : FEATURE = -single 

HasA Constituents: 

nOataPoint : DataPoint 

SingleValue 

A single value is a single value in time. 
Derived from IndividualValue 
Public Properties: 

pFeature : FEATURE = ^single 
HasA Constituents: 
a ValueLabel : ValueLabel 
aCategoryLabel : Category Label 
aInfoLabel : InfoLabcl 
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UlPrimitives 

The UlPrimimes care gory holds (he specialization hierarchy for the rerminal elements in a Ul: a 
pushbutton at the next higher ( U (Component) levei is nto primitives: the hotspoi and the label. 



Locatorfeutton 

A-locator~button-is activated-through-a-reiative-position-change'. or a click- hold-drag gesture. 

The input of a locator button is a relative change in either I. 2, or 3 dimensions. 
Derived from DirectionSelecior 
Public Properties: 

pFeature : FEATURE = -specific 



SelectorButton 

A selector button is a user-selectable Ul object that displays momentary (rather than 
persistent) action. Commonly used for commands ( "Print", either as a pushbutton or a menu 
item), menu access buttons, representation of process objects, and as the number ponton of a 
numbered verbal menu. 
Derived from [ndicationSelector 
Public Properties: 

pFeature : FEATURE = -changestate 
SpinnerButton 

A spinner button is a direction selector which is activated differentially depending on the 
amount of time between the activation and the deroctivation (commonly a click-hold-release 
operation in a direct manipulation interface). Spinner buttons are usually used in conjunction 
with an editbox to "spin" through a set of contiguous values (weekdays, numbers, etc.). 
Derived from DirectionSelecior 
Public Properties: 

pFeature : FEATURE = +specific 
vDirection : DIR = Up 

If the vName of me contains the name of a specialization of the DIR 
datatype, set the value of vDirection to the name of that DIR 
specialization class. 

Private Methods: 
SetDirection () : 

If the vName of me contains a defined DIR class, then set the vDirection of me lo the 
name of that class. 



OnOffButton 

An on-off button is an indication selector that displays persistent (rather than momentary) 
selection action. Used for states that change when the on-off button is selected, most 
commonly for visually presented check buttons and radio buttons. In an auditory interface, 
would be represented by Yes-No or True/False questions. 
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Denved from [ndicaiionSelector 
Public Properties: 
vState : BOOL = 0 

pFeacure : FEATURE = +changestate 

Pnvate Methods: 

SetState () : . , 

vStaie = True if my vUIName = vValue of my vReferent. 

TextEnter 

A string-editable U! object. The Text Enter usually has a related primitive Labeller, which 
displays the value of the Text Enter string. The text enter can be masked; v.hen masked, only 
mask-allowed entries can be made. Changes to the text enter are reflected by the associated 
Labeller value, 
Denved from Enterer 
Public Propenies: 
vMaskVaiue : STRING = 

The value of the mask string for this text box. 

pFeature : FEATIRE = +freetcxt 

Private Methods: 
SetMaskValue () : 

vMaskValue = vMask of my vReferent. 

Separator 

A separator is usually a line object that separates rwo groups of Ul objects: most commonlv 
the line between related sets of menu items in a menu list. 
Derived from Relator 
Public Propenies: 

vSeparatedltems : PAIR(UIOBJ) = 
pFeature : FEATURE = -continuity 

Pnvate Methods: 
SetSeparatedltetns () : 
Undefined. 

Grouper 

A grouper visually or temporally contains a number of other Ui objects to separate them from 
other Ul objects. Groupers include presentation of one menu node during a voice menu 
interaction, or a box around a set of mutuallv exclusive radio buttons. 
Derived from Relator 
Public Properties: 

vGroupedltems : LIST(UIOBJ) = ' 
pFeature : FEATURE = +union 

Private Methods: 
SetGroupedltems () : 

vGroupedltems = ail of the vOwns of the vQwner of me, except me. 



A hotspot IS a marker for something selectable (versus something "typable" or something that 
can only express information, not collect it as input). A hotspot can resolve to a menu item 
that can be voice selected, a pushbutton on aVDU.a cursor in a three dimensional space. 
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Derived trom UlPrimiiivc 
Public Properties: 

pFeature : FEATURE = taction 
Informer 

.Art informer is a category of U! primitive that presents information, but does not allow 
seiectionof ormodifxcation-oftheinformation. — 



Derived trom UlPrimitive 
Public Properiies: 

pFeature : FEATURE = +information 
Enterer 

An enterer is a basic Ul primitive that collects string information, as opposed to the other rwo 

classes off basic primitive that accept selection input or accept no input at all. 

Derived from HoiSpoi 

Public Properiies: 

vEditable : BOOL = 1 

vOptional : BOOL = 1 

pFeature : FEATURE = +keyboardcharacters 

Private Methods: 
SetEditable () : 

vEdiiable = vEditable of my vReferent. 
SetOptional () : 

vOptional = opposite(pRequired) of my vRrefereni. 

DirectionSelector 

A direction selection allows a direction to be indicated, either through a drag or through a spin 
most commonly, but could be through a joystick, a head pointer, or a v^^ink and a nod. for 
example. 

Derived from PushButton 

Public Properties: 

vincremcnt : INTEGER = I 

pFeature : FEATURE = ^amount 

Private Methods: 
Setlncrement () : 

vlncremeni = the vlncrement of my vReferent. 

IndicationSelector 

An indication selection accepts momentary selection input. 
Derived from PushButton 
Public Properties: 

pFeature : FEATURE = ^choice 
Axis ^ ^ .... 

An a.xis presents a senes of contiguous values arranged as a bounded line. 
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Derived from Relator 
Public Properiies: 

vOrientation : AXIS = Horizontal 
pFeature : FEATURE = ^continuity 

Pnvaie Methods: 
SetOrientatton () : 

[f the vUlName of me doesn't contain "Time", then set the vOrientation of me to 

Vertical. 

Labeller 

A labeller is non-selectable text that either presents static information or presents the name, 
type, or value of an associated dynamic Ul object (e.g,. the label on a button). 
Derived from Vajuelnformer 
Public Properties: 
vLabelData : STRING 

The actual content of the label. 

vAssociate : UIOBJ 

The associate is the UlPrimitive that is "associated" with this label; the 
hotspoi button that is labelled "Print", for example. 

pFeature : FEATURE = ^identity 

Pnvate Methods: 
SetLabelData () : 

The vLabeldaia is the vValue of the vReferent. 
SetAssociate () : 

The V Associate = vUIName of the hotspot that this labeller provides the label for. For a 
value label of a radio button, the associate is the button's vUIName; for the category label 
of the radio button group, it is.the vUIName of the pafeni (the vOwns of the labeller). 

DataPoint 

A data point is a point drawn to show a value at a time. Serves as the points on a data graph 

that is bound by two axes. ' 

Derived from Valuelnformer 

Public Properties: 

vDataValue : REAL 

vDataStatus : STATUS = Good 

vTimeValue : TIME 

pFeature : FEATURE = -t-number 

Private Methods: 

SetDataValue () : 

The vDataVaiue represents the value half of a time-value pair. 
SetDataStatus () : 

The vDaiaStatus is calculated by comparmg to the vAlarm and vWaming values of the 

referent for current values; for time-value pairs, the goodness is either an attribute of the 

time-value pair or calculated from other stored attribuies. 
SctTimeValue 0 : 

The limevalue is the time portion of a time-value pair. 

ViewLayout 

A viewlayout is a description of a visual presentation, usually a pointer to a file containing 
pointers to objects and a two or three dimensional representation of their physical 
relationships; may be a schematic drawing, a tagged Visiofile. or some other representation. 
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Denved from Informer 
Public Properiies: 

vFileName : FILENAME = V" 

filename of the drawing of the. contents of the display for this process 
group, drawing contains references to children of this process group, as 
objecinames and locations 

pF.eature_:_FEAT_URE_=_+context '. ^ 

Private Methods: 
SetFileName () : 

The vFilename of the view layoui is the Context Representation associated with the 
V Referent. 

Relator 

Derived from Informer 
Public Properiies: 

pFeature : FEATURE = +relationship 

Valuelnformer 

Derived from Informer 
Public Propenies: 

pFeature : FEATURE = rvalue 



PushButton 

Derived from HotSpot 
Public Properties: 

pFeature : FEATURE = +pick 



ValueLabel 

Value labels present a value for an object: the text in an edit box, the speed of a fan, etc, 
Denved from Labeiler 
Public Properties: 

vDataStatus : STATUS = Good 
pFeature : FEATURE = +status 

Pnvatc Mclh(xls: 

SetDataStatus () : 

The data status is calculated as a function of whether the vValue of the vReferent is 
greater than the pWamHt of the referent (STATUS = Warn) or greater than the pAlarmHi 
of the referent (STATUS = Alarm) or lower than the corresponding Lo warning and alarm 
values. 

CategoryLabel u f 

Category- labels label the category of an object: the "Color: " label for a setting choice, for 

example 

Derived from Labeiler 
Public Propenies: 

pFeature : FEATURE = +caiegory 



SUBSTITUTE SHEET (RULE 26) 



wo 99/34284 



-.44- 

LOGICAL VIEW REPORT 



PCT/US98/25968 



InfoLabel 

T\p€ labels present the type of an object: the shape of a fan icon, for^ example. 
Denved trom Labeljer 
Public Properties: 

pFeature : FEATURE = +into 

UlPrimitive 

Denved from UIOBJ 
Public Properues: 

pFeature : FEATURE = +primitive 

Public Methods: 
Setlndtvidual 0 : 

The public operation thai includes the private operations of the individual UlPnmitive. 



DOMAINCLASSES 



APPOBJ . . 

An appobject is an object that InFac can treat as if it is a real-world object tn the domain of 
interest. 

Public Properties: 
vName : STRING 

The name of an appobject is a unique identifier for that object. 

vLabel : STRING 

The label for an appobject is a user-understandable tag for the real-world 

appobject. 

vParent : APPOBJ 

The parent of an appobject is its container; a field on a SARIR has the 
set as its parent; the set has the SARIR. A 

vFeatures : HST(FEATURE) 

The features of an appobject are generated when InFac places it m it's 
domain model, or interacts with an object in another domain model. 
These features are used to select DIG objects to represent the appobject 
to a human- 
Public Methods: 

SetFeatures {argname : argtype = default) : return 

When an appobject is created for use by InFac, it sets its vFeatures attribute to a hsl ot ^ 
all the features that match the object's information properties. Currently defined as a walk 
through each branch of the features tree; for each category or feature class, the name of the 
feature is added to the list of vFeatures if the appobject's associated field (as defined for 
each feature) matches the feature's parameters. 

SetOtherVars (argname ; argtype = default) : return 

HasA Constituents: 

aContextReprcsentation : ContextRepresentation 
aGraphicRcpresentation : GraphicRepresentation 
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DomainComponent 

A domain component is an object in the domain which is make up of other components or 
domain primitives. 
Derived from APPOBJ 
HasA Constituents: 

nConiainedObjects_:_ABBOBJ 

DomainPrimitive 

A domain primitive cannot contain other appobjects: A ship is a domain component which 

has at least one domain primitive, shipcolor. ShipCoior is also contained by SEAINCDT. a 

set in a SARfR, Other examples are Person 'sName. BusinessAddress. NumberOfFans. 

Domain components at any level can have domain primitives: domain primitives cannot be 

further decomposed. 

Derived from APPOBJ 

Public Propenies: 

vValue : STRING . , 

The value of a domain primitive is the actual value of the reaiworld 
appobject. The pValue of the domain primitive shipcolor = red; of the 
domain primitive fan speed = 100%. 

vRequired : BOOL 

Whether or not the domam primitive is required to have a value. 

vSecrecy : INTEGER 

Level of secrecy of the value of the domain primitive. 

vSource : STRING 

Source of the value of the domain primitive. 

vEditable : BOOL 

Whether or not the value of the domain primitive is editable (regardless 
of secrecy level). 

Ha.sA Constituents: 
aScheduie : Schedule 
aValueHistory : ValueHistory 

ContinuousDataltem 

A data item whose value is a point on a continuous scale. These are traditional seipotnis am 
sensor values. Specialized as Duration, Speed. Temperature. Quantitiy. 
Derived from ValueDaialtem 
Public Properties: 
vVfinimum : REAL 

The logial minimum for the data item. 

V Maximum : REAL 

The logical maximum for the data item. 

vAlarmHi : REAL 

The high alarm value; the value of the data item at which a high alarm 
value is generated. 
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vAlarmLo : REAL 

The lower bound value of the. data, item at which an alarm is generated, 

vWarnHi : REAL 

The high bound value of a data item at which a wammg is generated. 

vW^rnLo : REAL 

The low bound value of a data iiem at which a wammg is generated. 

V Value : REAL 

The current value of the data item. 

vUnits : MEASURE = NULL 

The units, if any. in which the value of the data item are expressed. 

V Requested : REAL • 

The requested (as opposed to the actual) value of the data item: e.g., the 
room temperature setting or the requested speed. 

vincrcment : INTEGER 

The logical increment for increasmg or decreasmg the requested or actual 
value of a dataitem. 

vSpecificity : BOOL = 1 

Whether or not the daiaitem is specific '(l=true) or relative (0=faise); 
captures distinction requiring an additional slider and axis if not specific. 



DiscreteDataltem 

A discrete data item is a dataitem that contains a. value that is one of a set of allowable values. 
The specializations of discrete data item are entity, state, property, action, cause. 
Derived from ConstrainedDataliem 
Public Propenies:. 
vExtendable : BOOL 

Whether or not the set of Choices for the data item can be extended. 

vOrdered : BOOL 

The number of values a data item can take; usually = I (exclusive item 
from a set), but may be greater than 1 for non-exclusive choices 
(members of a team, choices from a three column choice). 

vNumChoices : INTEGER 
UniqueDataltem 

A unique data item is one whose value is not a numeric value and not a member of a set. It 
has the specializations of Unique Name, Unique Number GenerallD and Descriptive Prose. 
Derived from DomainPrimitive 

ValueHlstory 

The value history is a history of the values of the appobject. 
Dcnvcd from DATA VIEW 
Public Properties: 

pTimeSpan : TIMESPAN = Past 



SUBSTITUTE SHEET (RULE 26) 



wo 99/34284 



PCT/US98/25968 



-47- 

LOGICAL VIEW REPORT 



Schedule 

The schedule provides the time -value or event- value pairs that make up the predicted and/or set 
values for the future behavior of the appobject. 
Derived from DATAVIEW 
Public Properties: 

pTimeSpan : TIMESPAN = Future 



GraphicRepresentation 

.A graphic representation provides the graphic for the object, often an icon. 
Derived from DATAVIEW 
Public Propeniesf 

pTimeSpan : TIMESPAN = Current 
DATAVIEW 

A data view is a file associated with an appobject that contains some aspect of its data. 

Public Properties: 

pFilename : FILENAME 

The filename assoicaied with daiaview is the name of the file that 
contains the appobject's data. , 



Action 

The data item's value represents an action: View Trend, Acknowledge 
Dcnvcd from DiscrctcDataltcm • . . 



State 

The dataitem 's value represents an object state, like on or sleeping. 
Derived. from DiscreteDataltem^ 



UniqueNumber 

A choice that represents a unique identifier, like asocial security number, a call stgn, a serial 

number, a filename, 

Denved from UniqueDataliem 

Public Properties: 

vMask : MASK 

Temperature r u- 

A temperature data item has a value that represents the temperature of something. 

Derived from ConcinuousDataltem 

Public Propenies: 

vUnits : MEASURE = degF 

The default measure for temperature is degF (degrees fahrenheit). 



Property 

The dataitem's value represents a property, like color. 
Denved from DiscreteDataltem 
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Speed 

Speed IS a data item representing a distance over time value (niph, rpm). 

Denved trom ContinuousDataliem 

Public Properties: 

vtnits : MEASURE = rpm 

The default measure for speed is rpm (revolutions per minute). 



DescProse 

A data item whose value is prose, generally a freetext description of something. 
Denved from UniqueDataliem 

GeneratID 

Derived from UniqucDaialtcm 



UniqueName 

A unique name is a type of data item whose value represents a name of an entity, like "Sam 

Jones " or Houston. 

Derived from UniqueDataltem 

DateTimeGroup 

A data item whose value represents a date and time, with year, month, day, hour, mtnute, 
second, msec. 

Derived from TimeDaialtem 
Public Properties: 
vVaiue : DATETIME 

DateValue 

A data item whose value represents a date, including day, month, and year 
Denved from TimeDataltem ^ ' - 

Public Properties: 
vValue : DATE 



MonthValue 

A data item whose value is the name of a month, 
Denved from TimeSet 
Public Properties: 
vValue : MONTH 

Cause 

The data item's value represents a cause, like fire. 
Derived from DiscreteDataltem 

^^^A^discrete data item whose value represents an entity or object; a motor may have one of three 
distinct flanges. 
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Derived from DiscreieDataltem 
TimeDataltem 

A timeitem data item is a data item whose value represents a time. 
Derived from ConstrainedDataltem 



Duration 

Duration is a data item that represents a span of time (2 hr. 3 yr 12 mos) 

Derived from ContinuousDaiaitem 

Public Properties: 

vUnits : MEASURE = min 

The default units for duration is minutes. 



ContextRepresentation 

A context representation is a file that provides the context for an appobject. 
Denved from DATA VIEW 

Public Properties: . . 

pTimeSpan : TIMESPAN = Current 



TimeValue 

A data item whose value is a time value, with hours (of 24), minutes, seconds, and posstbly 
msec. 

Derived from TimeDataltem 
Public Properties: 
V Value : TIME 



ValueDataltem 

.4 value data item is a dataitem whose value is a numeric value, either a timeitem. a location, 
or a continuous data value. 
Denved from ConstrainedDataliem 



Location 

A data item whose value represents a location. 

Derived from ValueDataltem 
Public Properties: 
vVaiuc : LOC 

Quantity „ x. 

A quantity data item has a value representing a quantity of something, usually with no units. 
Derived from ConiinuousDataltem 



Quality 

Derived from DiscreteDataltem 
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SingleAction 

Derived from UniqueDaialiem 

SingleEntity 

Derived from UniqueDataltem 

TimeSet 

Derived from Discrete Dataliem 

ConstrainedDataltem 

Derived from DomainPrimitive 
Public Properties: 
vChoiccs : LIST 

VARTYPES 



VariabieType 

The root node for variable types for DIG and for the Domain 

Common VariableTypes 

Variable types that are common to all objects. 
Denved from VanableType 

BOOL 

Boolean: 0~ False, I^True 

Derived from Common VariableTypes . 

STRING 

A character string: 

Denved from Common VanableTypes 
CONTEXT 

A context is the scope of reference of an object: an object applies, for example, to an 
application, or an object within an application, or a value of a property of an object in an 
application, etc. 
Derived from OrCVariableType 

FORMAT 

The prototypical format for a task or interaction. 
Denved from DIG VariabieType 

FILENAME 

Pointer or reference to a file location. 
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Derived from CommonVariableTypes 
INTEGER 

A v\'hoie number: may be negative. 
Derived from CommonVariableTypes 

REAL 

A regular old number with a decimal and/or a negative sign. 

Derived from CommonVariableTypes 

TIMESPAN 

A timespan reflects that origin of information or the location in time of an action: current , 
information, past information, future (planned or projected) information. 
Derived from CommonVariableTypes 

Past 

Timespan that has occured. 
Derived from TIMESPAN 

Current 

Ongoing timespan. 

Derived from TIMESPAN „ ■ ^ 

Future 

Timespan that has not occured. 
Derived from TIMESPAN 

DIGVariableType ' . * , mr 

The root variable type. Variable types are used to type the attributes of objects in the UIO 
model. 

Dcnved from VariableType 
APP 

The context of an application. 
Derived from CONTEXT 

OBJ 

The context of an object within an application. 
Dcnved from CONTEXT 

Choices 

A set of choices. 
Derived from FORMAT 
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Form 

A form format, where an object is the subject of the form, and the various relevant properties 
or attributes of the object are presented as labelled fields containing the value of the labelled 
property. >. - - / 

Dcnved from FORMAT . 

Trend 

A value over time graph. 
Derived from FORMAT 

Map 

A map or schematic format, where the object is represented by the (ZD) map and the sub- 
objects are shown in physical relationship to each other. 
Derived from FORMAT 

RESULT 

A result marks the message sent back from a dynamic component. 
Denved from DICVanablcType 

ChangeValue 

The result of invoking an object is to change the value of that object. 
Derived from RESULT 

InvokeAction 

The result of invoking an object is to invoke the action it represents. 
Denved from RESULT 

SelectObject 

The result of invoking an object . is to select the object it represents. 
Denved from RESULT 

Selectltem 

The result of invoking a dynamic object is to select that object. 
Denved from RESULT 

ExclusiveSelectltem 

The result of invoking an object is to select that object and de^select all other objects in its 
group. 

Derived from RESULT 

ChangeState 

Changes the state of the object. 
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Denved from RESULT 
DIR 

Dir is a direction: up, down, right, left; in relation to the viewer. 
DcrivciJ from DIGVariablcTypc 

•Up ^ — - 

Up against gravity. 
Derived from DIR 

Down 

Down towards gravity. 
Derived from DIR 

Right 

Right in relation to the viewer. 
Derived from DIR 

Left 

Left in relation to the viewer. 
Derived from DIR 

Domain VariableTypes 

The root node for the domain-specific variables: these variables and datatypes are used by the 
appobjects and not the uiobjectSi 
Derived from VariableType 

LOG 

This is an abstract data type that represents a location. 
Derived from DomainVariableTypcs 

TIME 

An abstract data type that represents a time value, includes hour (24), mm, sec, msec. 
Derived from DomainVariabieTypes 

DATETIME 

A date time value, with year, month, day, hour (24), minute, sec, msec. 
Derived from DomainVariabieTypes 

DATE 

A date value: year, month, day 
Derived from DomainVariabieTypes 

MONTH 

.A month value. 
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Denved from Domain VahableTypes 
MASK 

A mask template, like nnn-nn-nnnn for social security number. 

Derived from Domain VanableTypes . . ; .. . *; . . 

STATUS 

The status is a valuation of the goodness of a piece of data: within or without the alarm or 

tolerance boundaries, 

Denved from DIGVariableType 

Good 

Derived from STATUS 

Warn 

Denved from STATUS 

Alarm 

Denved from STATUS 



AXIS 

The two-dimensional axis. 
Derived from DIGVariableType 

Horizontal 

Horizontal (left-nght) axis. 
Denved from AXIS 

Vertical 

Vertical (up-down) axis. 
Denved from AXIS 



MEASURE 

A measure data type represents the various types of units which express a i 
Denved from DomainVariablcTypcs 

SpeedMeasure 

Speed measures are units in which speed may be expressed 
Denved from MEASURE 

rpm 

revolutions per minute is the default speed unit of measure. 
Denved from SpeedMeasure 
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mph 

miles per hour is a unit of speed measure menu 
Derived from SpeedMeasure 

TemperatureMeasure 

Temperature measures are units in which temperature may be expressed, 
Derrved-ffoTTTMEASURE 

degF 

degrees Fahrenheit is the default temperature measure. 
Derived from TemperatureMeasure 

degC 

degrees Centigrade is a common temperature measure. 
Derived from TempeniiureMeasure 

DistanceMeasure 

Distance measures are units in which distances or lengths may be expressed. 
Derived from MEASURE 

mile 

mile is the default distance measure. 
Denved from DistanceMeasure 

TimeMeasure 

Time measures are units in whih time may be expressed. . 
Derived from MEASURE 

min 

min is the default time measure 
Denved from TimeMeasure 

FEATURE 

A feature distinguishes an object from the other objects at it's level of detail. 
Derived from DlGVuriableType 

taction 

The object represents/doesn 't represent an action. 
Derived from FEATURE 

•^amount 

Data reflects an amount. 
Derived from FEATURE 
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^application 

The object is an application. 

Derived trom FEATURE 

+category 

Category of an object. 

Derived from FEATURE 

H-changestate 

Object changes a state. 

Derived from FEATURE ^ 

+cholce 

Action is choice. 
Derived from FEATURE 

+comtnunication 

Denvcd from FEATURE 

-^component 

A feature marking a component in the Interaction- Space. 
Derived from FEATURE 

^context 

Derived from FEATURE 

-hcontinuity 

Data does or does not reflect a continuity. 
Derived from FEATURE 

+dennition 

Provides definition information. 
Derived from FEATURE 

+discourse 

This feature discriminates between discourse and non-discourse elements. 
Derived from FEATURE 

+discrete 

Data is discrete. 
Derived from FEATURE 

^dismiss 

Dismisses another object. 
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Derived from FEATURE 

+dynami€ 

Derived from FEATURE 

^equivalence 

Equivalence data 
Derived from FEATURE 



+exclusive 

An exclusive choice from a set. 
Derived from FEATURE 



+ fixed 

Data set is fixed 
Derived from FEATURE 

+hierarchical 

Data is organized hierarchically rahter than in in a single list. 
Derived from FEATURE 

+identity 

Data provides identity information. 

Derived from FEATURE 



+ info 

Information about an object. 

Derived from FEATURE 
+information 

Data that provides information. 
Derived from FEATURE 

^invoke 

The object invokes/ another object. 
Derived from FEATURE 



+itein ^ ^ 

The object is/ is not a sub^item (property) of another object. The color of an object is ^item. 
the various parts of an object are -t-subobject. 
Derived from FEATURE 



-hkeyboardcharacters 

Data is composed of keyboard characters. 
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Derived from FEATURE 
+meta 

The object is/is not a meta object. 
Derived from FEATURE 

^number 

Derived from FEATURE 

+object 

This feature denots an object. 
Derived from FEATURE 

+objects 

Derived from FEATURE 

+overtiine 

No state change 
Derived from FEATURE 

+pick 

Action is pick. , , , 

Derived from FEATURE 

+pian 

Provides plan information. 
Derived from FEATURE 

^primitive 

Denved from FEATURE 

^relationship 

Data reflects a relationship. 
Derived from FEATURE 

-i-review 

A review feature indicates whether the information is a summary/review or 
Denved from FEATURE 

+set 

An item of a sett. 
Derived from FEATURE 
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^-single 

Whether or not an element is a single element or a group. 
Derived trom FEATURE 

4-specific 

A specific, rather than a continuous, value, 

DerivedfromFEATURE 



estate 

A state component. 
Derived from FEATURE 



^Status 

This feature denotes a status object. 
Derived from FEATURE 

-hsubobject 

The object does not represent a value. 
Derived from FEATURE 

+subtask 

The object is/is not a subobject (part) of it's parent object. 

A steering wheel is a ^subobject of a car: a passenger is -subobject. The color of the car is 
■^item. 

Derived from FEATURE 
+systemtouser 

The element involves information transferred from a system to a user 
Denved from FEATURE 

+task 

The object is a task. 
Derived from FEATURE 



+ttme 

The object represents/doesn't represent a time value. 
Derived from FEATURE 



atopic 

The object is/is not a topic of a higher level object. 
Derived from FEATURE 



^union 

Relationship is one of union. 
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Denved from FEATURE 
-f-usertosystem 

The element involves information transfer from a user to a system, 
Denved from FEATURE 

rvalue 

The object represents a value. ; ; 

Derived from FEATURE - 

+ view 

Derived from FEATURE 

+ views 

The object is/ is not a view of another object. 
Denved from FEATURE 

-continuity 

Denved from FEATURE 

-discrete 

Data is continuous. 
Denved from FEATURE 

-dynamic 

Denved from FEATURE 

-exclusive 

A non-exclusive choice. 
Derived from FEATURE 

-fixed 

Data set is extendable. 
Derived from FEATURE 

-hierarchical 

A single rather than hierarchical list. 
Denved from FEATURE 

-set 

Derived from FEATURE 
-single 

Not a single element, but a group. 
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Derived from FEATURE 



• specific 

Not a specific, but a continuous, value. 

Derived from FEATURE 



INTERACTIONMANAGEMENT 

The InteractionManagement category is outside of DIG. it is the category that holds the agents and 
techs and any other information, models, or heuristics required to perform Dynamic Interaction 
Generation. 



InteliigentEntitites 

SocietyAgent 

Derived from InteliigentEntitites 

InteractionFacilitator 

The Interaction Facilitator (InFac) tells a class in the User Interface Model to create itself as a 
named e 

Derived from SocietyAgent 
Technician 

Derived from InteliigentEntitites 

JavaTech 

The JavaTech is the technician who knows how to.translate the objects tn the Interaction- 
Space into Java.awt objects and present them on the screen. 
Derived from Technician 
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What is claimed is: - . ; , 

1 . An interaction synthesizing system for a computer running an application 
program formed of multiple objects, the system comprising: 

an interaction space corhposed of multiple design elements corresponding to 
active application objects; and 

' an interface synthesizer which modifies the interaction space in response to 
changes in the application objects. 

2. The system of claim 1 wherein the application objects are representative of 
sensed parameters. . , 

3. A method of creating a user interface for an application program running on a 
computer system having devices available for commimicating with a user, comprising 
the steps of: - 

retrieving an interaction model of interaction requirements for general situational 
components of the application program; 

creating an interaction space representative of current interaction requirements of 
the application program using the interaction model; and 

dynamically creating a user interface for the application program based on the 
interaction space and available user interaction devices. 

4. The method of claim 3 wherein the available interaction devices have different 
capabilities for interacting with a user, and wherein the user interface is dynamically 
created responsive to those different capabilities. 

5. The method of claim 4 wherein one of the devices comprises a telephone, and 
the other device comprises a computer, and wherein the interface created for the 
telephone provides information to the user in the form of voice, and wherein the 
interface created for the computer comprises a visual representation of the interaction 

0 space. 
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6. A method of creating an interaction space having components representative of 
user interaction needs with an application program, comprising the steps of: 

perceiving an interaction need; 

entering an interaction class^model at a point dependent on the received 
5 interaction need; 

specializing the class model to arriye at an instantiable node in the interaction 
class model; and 

adding the instantiable node to the interaction space. 

7. The method of claim 6 and further comprising the step of: 
converting the interaction space into a user interface. 

i' ' • 

8. The method of claim 7 wherein the conversion of the interaction space into a 
user interface step is performed as a function of capabilities of abuser interaction device, 

9. An interaction class model used to dynamically generate user interaction 
representations for a current state of an application, the model comprising: 

a top level abstract user interface object having a create operation; 
multiple intermediate level abstract objects representing different abstract user 
interaction types, and inheriting capabilities from the top level user interface object; and 

multiple instantiable objects representative of user interface elements, and that 
inherit capabilities from the top level and intermediate level objects. 

10. The class model of claim 9, wherein the multiple intermediate level abstract 
25 objects comprise objects selected from the group consisting of tasks, subtasks, 

components, and primitives. 

11. A machine readable medium having instructions stored thereon for causing a 
suitably programmed computer to perform the steps comprising: 

30 retrieving an interaction model of interaction requirements for general situational 

components of the application program; 
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. • constructing a representation of current-interaction requirements of the 

application program using the model; - 

creating an interaction space froni the. representation and abstract compositional 
constraints and selected properties of the application program; and 
5 dynamically creating a user interface for the application program based on the 

interaction space and available devices. 

12. A machine readable medium having instructions stored thereon for causing a 
suitably programmed computer to perform the steps comprising: 

10 perceiving a change in an object in an application having an interaction need; 

entering an interaction class model at a point dependent on the perceived 
interaction need; 

specializing the class model to arrive at an instantiable object in the interaction 
class model; and 

15 adding the instantiable object to the interaction space. 

13. A system comprising: 

an application comprised of multiple objects, including objects for interacting 
with users; 

20 an interaction class model having multiple instantiable objects; and 

an interaction facilitator coupled to the application and to the interaction class 
model for creating an interaction space comprised of instantiated class model objects 
responsive to currently active application objects. 

25 14. The system of claim 1 3 wherein the application comprises multiple dynamically 
varying networked objects. 

15. The system of claim 13 wherein the application is selected from the group 
consisting of search and rescue applications, building management applications, and 
30 process control applications. 
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16. The system of claim 13 and further comprising a module coupled to the 
interaction space for converting interaction elements to a user interface on a selected 
device based on the capabilities of the selected device. 

5 17. The system of claim 16 wherein. the module is selected from the group 
consisting of a JavaTech, and a SpeechTech. 
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